Support async networking #63

rnburn · 2020-04-16T22:29:42Z

Adds an interface for managing async file and timer events within the io polling system calls.

Will support asynchronous networking for tracers and meters.

A concrete implementation is provided with libevent.

The dispatcher interface was derived from Envoy.

pyohannes · 2020-04-17T00:10:41Z

Will support asynchronous networking for tracers and meters.

What major use cases for asynchronous networking do you have in mind? Is this intended to be used by exporters?

rnburn · 2020-04-17T02:44:22Z

Asynchronous networking is more efficient. If the tracer uses synchronous networking, then it could be blocked writing data to a socket even if could move forward with work on other sockets or events.

Exporters can use the interface, but they're also welcome to do their own synchronous networking (though that will block the background thread).

Additionally, asynchronous networking is important for allowing an application to exit quickly. If you do a synchronous write to an endpoint that's not responding and the app exits, you can hang up the app until the write's timeout is exceeded.

pyohannes · 2020-04-17T03:18:21Z

Asynchronous networking is more efficient. If the tracer uses synchronous networking, then it could be blocked writing data to a socket even if could move forward with work on other sockets or events.

Yes, I completely agree. I mostly wonder where in the SDK it will be used, apart from exporters.

rnburn · 2020-04-17T03:32:47Z

It could also be used in the sdk to set up the the timers that regularly trigger the span buffer to flush (similarly for metrics)

And asynchronous dns resolution can be supported over an interface like this with c-ares.

g-easy · 2020-04-20T03:46:05Z

bazel/libevent.BUILD

+    }),
+    make_commands = select({
+      "@io_opentelemetry_cpp//bazel:windows": ["MSBuild.exe INSTALL.vcxproj"],
+      "//conditions:default" : None,


Looks like this file didn't go through buildifier.

Update format script to run on these files

Done in #69

ci/install_libevent.sh

sdk/include/opentelemetry/sdk/event/dispatcher.h

g-easy · 2020-04-20T03:48:48Z

sdk/include/opentelemetry/sdk/event/dispatcher.h

+  /**
+   * Run the event loop until.
+   */
+  virtual void Run() noexcept = 0;


Where is this documented? As a user of the SDK, what do I need to do with the Dispatcher?

It's intended to support a design where you have a single background thread and asynchronous networking. The background thread would be making successive calls to the systems multiplexing io function (e.g. epoll, kqueue, etc) and invoking callbacks as events are ready.

If you're a user of the SDK and you wanted to use async networking for your vendor exporter, you'd call CreateFileEvent and pass the file descriptors for the sockets you used to connect to your endpoint.

Or if you don't want to do asynchronous io, you can ignore the dispatcher, use synchronous sockets, and block the background thread when you want to do io (less efficient and can hang the process from exiting but supported).

Can we capture this in a doc please? :)

Added commenting

sdk/include/opentelemetry/sdk/event/file_event.h

sdk/src/event/libevent/BUILD

reyang · 2020-04-27T21:08:56Z

sdk/test/event/libevent/io_dispatcher_test.cc

+  t1 = std::chrono::steady_clock::now();
+  dispatcher.Run();
+  auto duration = t2 - t1;
+  EXPECT_TRUE(duration > std::chrono::milliseconds{50});


On different CI systems we might need to adjust the tolerance of timing.
Consider making this a configurable thing which we could leverage in other test cases (in particular metrics and async exporter).

https://github.com/census-instrumentation/opencensus-python/blob/99821a0b826445cecce0881a01790c8cc6958ceb/tests/unit/metrics/test_transport.py#L32

I tried to pick a constant that should be large enough for most environments, though we can probably go larger without slowing down the tests too much.

reyang

LGTM.

g-easy · 2020-05-04T07:24:15Z

Just needs an autoformat to pass CI.

g-easy · 2020-05-11T07:23:45Z

sdk/src/event/async_timer/dispatcher.cc

-void Dispatcher::Exit() noexcept {}
+void Dispatcher::Exit() noexcept
+{
+  running_ = false;


Not meant to be threadsafe?

Yes, meant for interacting with only a single background thread.

g-easy · 2020-05-11T07:25:06Z

sdk/src/event/async_timer/dispatcher.cc

+  while (running_ && !events_.empty())
+  {
+    auto next_event = events_.begin();
+    std::this_thread::sleep_until(next_event->first);


How can this sleep be interrupted?

I'd recommend a short polling interval over an interrupt mechanism.

I'd rather people use the libevent implementation, and this is meant to be a drop-in replacement that provides the same interface and allows you to use the same codebase but without an event library (in case you only care about synchronous networking and don't want to add an external dependency).

When the dispatcher is built on top of libevent, the background thread makes successive calls to the systems polling function (e.g. epoll or an equivalent). Epoll blocks waiting for either a timeout or and event on a list of file descriptors.

I believe it is possible to artificially interrupt epoll if you create a dummy file descriptor for it to include in its list and then close the file descriptor, forcing an event.

But I prefer, and I think it's simpler, to use a polling function with a short timer interval that monitors for termination, closure, and flush events. Basically what's done here
https://github.com/lightstep/lightstep-tracer-cpp/blob/master/src/recorder/stream_recorder/stream_recorder_impl.cpp#L57-L82

sleep_until may hang "forever" in gcc-4.8 and vs2017 15.4 when system clock changes, i.e. after NTP time sync.

It uses the steady clock which shouldn't have that problem

it is possible interrupt epoll

According to man epoll_wait can be interrupted by a signal handler;

you create a dummy file descriptor for it to include in its list and then close the file descriptor, forcing an event.

ref: http://cr.yp.to/docs/selfpipe.html

codecov · 2020-07-07T22:18:42Z

Codecov Report

Merging #63 (cc34479) into main (65f6639) will decrease coverage by 2.90%.
The diff coverage is 54.28%.

@@            Coverage Diff             @@
##             main      #63      +/-   ##
==========================================
- Coverage   93.61%   90.70%   -2.91%     
==========================================
  Files          71       85      +14     
  Lines        1754     1894     +140     
==========================================
+ Hits         1642     1718      +76     
- Misses        112      176      +64

Impacted Files	Coverage Δ
sdk/include/opentelemetry/sdk/event/file_event.h	`0.00% <0.00%> (ø)`
sdk/src/event/libevent/file_event.cc	`0.00% <0.00%> (ø)`
sdk/src/event/libevent/file_event.h	`0.00% <0.00%> (ø)`
sdk/src/event/libevent/io_dispatcher.cc	`50.00% <50.00%> (ø)`
sdk/src/event/libevent/event_base.cc	`54.54% <54.54%> (ø)`
sdk/src/event/libevent/event.cc	`65.62% <65.62%> (ø)`
sdk/src/event/libevent/timer.cc	`76.92% <76.92%> (ø)`
sdk/test/event/libevent/io_dispatcher_test.cc	`85.71% <85.71%> (ø)`
sdk/include/opentelemetry/sdk/event/dispatcher.h	`100.00% <100.00%> (ø)`
...dk/include/opentelemetry/sdk/event/io_dispatcher.h	`100.00% <100.00%> (ø)`
... and 4 more

pyohannes · 2020-10-22T22:27:14Z

Assigning to @lalitb, as discussed in the SIG meeting on October 21st.

The main challenge here would be rebasing and making all GH actions pass.

alolita · 2020-12-07T23:19:16Z

Triaged Dec 7 2020: This is a good change to integrate but non-trivial. The maintainers need to figure out what items are blocked on this. This is not an immediate priority. Will keep open until re-evaluated.

ThomsonTan · 2021-09-24T07:23:41Z

@lalitb do we need rework on this PR to get it merged?

linux-foundation-easycla · 2021-12-15T01:50:59Z

✅ Ryan Burn (73d9d9d, 9685fb2, d6b175f, 2b86e11, 8f387dc, bd57639, 3448df0, 2d2ea8e, dc35d6e, fb83a8d, 2d40011, cabab7c, d76fd24, fda54ee, f3bb9b9, e5596d8, fe208d1, 3ac2aa5, 919d396, 03eb077, 7d343ce, 7bbbe29, 11f0e46, 515d01c, 18791c1, 68ce934, 72fd4df, 30cff79, e79b5f6, 9b62085, c120b8d, 61b6b94, 29a6b2f, 7b59314, 23009af, d14c11f, 0cef1d8, 8c2b945, 2f0cf96, 76683b0, 53bcda4, 4b162e4, 3212ed4, 8f358c9, e086e77, ea0e87c)

❌ The commit (6234ee6 ,faf297319f1d0947ae74b719564ed25f1828247e ,1de581b781dc07ffe97650aa5fd9b5ed9d84692c ,2cbf66ca2938df2326465e07999d30b8cc227275 ,8248ddd574bdec2eb7db0fc4f870457cf01230e1 ,708f7b0c483ca2713320f66595626e5403ca5337 ,cc34479373512e7090a706a76fa843af75af8cbe) is missing the User's ID, preventing the EasyCLA check. Consult GitHub Help to resolve.For further assistance with EasyCLA, please submit a support request ticket.

lalitb · 2021-12-22T02:05:18Z

Keeping it open, for someone to take a dig at it.

linux-foundation-easycla · 2021-12-22T02:05:26Z

✅ Ryan Burn (73d9d9d, 9685fb2, d6b175f, 2b86e11, 8f387dc, bd57639, 3448df0, 2d2ea8e, dc35d6e, fb83a8d, 2d40011, cabab7c, d76fd24, fda54ee, f3bb9b9, e5596d8, fe208d1, 3ac2aa5, 919d396, 03eb077, 7d343ce, 7bbbe29, 11f0e46, 515d01c, 18791c1, 68ce934, 72fd4df, 30cff79, e79b5f6, 9b62085, c120b8d, 61b6b94, 29a6b2f, 7b59314, 23009af, d14c11f, 0cef1d8, 8c2b945, 2f0cf96, 76683b0, 53bcda4, 4b162e4, 3212ed4, 8f358c9, e086e77, ea0e87c)

❌ The commit (6234ee6 ,faf297319f1d0947ae74b719564ed25f1828247e ,1de581b781dc07ffe97650aa5fd9b5ed9d84692c ,2cbf66ca2938df2326465e07999d30b8cc227275 ,8248ddd574bdec2eb7db0fc4f870457cf01230e1 ,708f7b0c483ca2713320f66595626e5403ca5337 ,cc34479373512e7090a706a76fa843af75af8cbe) is missing the User's ID, preventing the EasyCLA check. Consult GitHub Help to resolve.For further assistance with EasyCLA, please submit a support request ticket.

rnburn requested review from g-easy, jmacd, maxgolov, reyang and tigrannajaryan as code owners April 16, 2020 22:29

g-easy reviewed Apr 20, 2020

View reviewed changes

reyang reviewed Apr 27, 2020

View reviewed changes

reyang approved these changes Apr 27, 2020

View reviewed changes

g-easy reviewed May 11, 2020

View reviewed changes

pyohannes mentioned this pull request May 11, 2020

Implement a batching span processor #83

Closed

pyohannes requested a review from a team July 6, 2020 23:04

rnburn added 14 commits July 7, 2020 09:45

Setup dispatcher interface

73d9d9d

Start libevent implementation

9685fb2

Add EventBase class

d6b175f

Integrate EventBase

2b86e11

Add Event

8f387dc

Add Timer

bd57639

Add Timer test

3448df0

Add FileEvent

2d2ea8e

Add Exit test

dc35d6e

Integrate libevent with cmake

fb83a8d

Drop kTimeout

2d40011

Integrat libevent with CI

cabab7c

Add commenting

d76fd24

Reformat

fda54ee

Update GH Actions

6234ee6

pyohannes force-pushed the network branch from d21c987 to 6234ee6 Compare July 7, 2020 17:07

Johannes Tax added 6 commits July 7, 2020 11:19

Fix CI

faf2973

Build examples only with SDK

1de581b

Add libevent

2cbf66c

Fix WITH_SDK flag

8248ddd

Increase timeout to avoid flaky tests

708f7b0

Fix Windows dependencies

cc34479

pyohannes assigned lalitb Oct 22, 2020

Base automatically changed from master to main January 26, 2021 00:46

lalitb mentioned this pull request Mar 22, 2021

Create SDK TRACE context, a shared reference to tracer pipelines between Tracer/TracerProvider. Update tests. #590

Closed

github-actions bot added the Stale label Dec 7, 2021

github-actions bot closed this Dec 15, 2021

lalitb reopened this Dec 15, 2021

github-actions bot closed this Dec 22, 2021

lalitb reopened this Dec 22, 2021

github-actions bot removed the Stale label Dec 23, 2021

lalitb removed their assignment Jan 26, 2022

lalitb mentioned this pull request Mar 3, 2022

Async/callback based interface between Span/Log Processor and Span/Log Exporter #1239

Closed

lalitb self-assigned this Mar 7, 2022

lalitb mentioned this pull request Mar 8, 2022

Add support for event driven framework #1250

Open

github-actions bot added the Stale label May 7, 2022

github-actions bot closed this May 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support async networking #63

Support async networking #63

rnburn commented Apr 16, 2020

pyohannes commented Apr 17, 2020 •

edited

Loading

rnburn commented Apr 17, 2020

pyohannes commented Apr 17, 2020

rnburn commented Apr 17, 2020 •

edited

Loading

g-easy Apr 20, 2020

rnburn Apr 22, 2020

g-easy May 4, 2020

g-easy Apr 20, 2020

rnburn Apr 20, 2020

g-easy May 4, 2020

rnburn May 11, 2020

reyang Apr 27, 2020

rnburn May 1, 2020

reyang left a comment

g-easy commented May 4, 2020

g-easy May 11, 2020

rnburn May 11, 2020

g-easy May 11, 2020

rnburn May 11, 2020 •

edited

Loading

maxgolov May 13, 2020

rnburn May 13, 2020 •

edited

Loading

amirouche Sep 27, 2023

codecov bot commented Jul 7, 2020 •

edited

Loading

pyohannes commented Oct 22, 2020

alolita commented Dec 7, 2020

ThomsonTan commented Sep 24, 2021

linux-foundation-easycla bot commented Dec 15, 2021

lalitb commented Dec 22, 2021

linux-foundation-easycla bot commented Dec 22, 2021

Support async networking #63

Support async networking #63

Conversation

rnburn commented Apr 16, 2020

pyohannes commented Apr 17, 2020 • edited Loading

rnburn commented Apr 17, 2020

pyohannes commented Apr 17, 2020

rnburn commented Apr 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reyang left a comment

Choose a reason for hiding this comment

g-easy commented May 4, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rnburn May 11, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rnburn May 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 7, 2020 • edited Loading

Codecov Report

pyohannes commented Oct 22, 2020

alolita commented Dec 7, 2020

ThomsonTan commented Sep 24, 2021

linux-foundation-easycla bot commented Dec 15, 2021

lalitb commented Dec 22, 2021

linux-foundation-easycla bot commented Dec 22, 2021

pyohannes commented Apr 17, 2020 •

edited

Loading

rnburn commented Apr 17, 2020 •

edited

Loading

rnburn May 11, 2020 •

edited

Loading

rnburn May 13, 2020 •

edited

Loading

codecov bot commented Jul 7, 2020 •

edited

Loading